63 research outputs found
Developing All-Skyrmion Spiking Neural Network
In this work, we have proposed a revolutionary neuromorphic computing
methodology to implement All-Skyrmion Spiking Neural Network (AS-SNN). Such
proposed methodology is based on our finding that skyrmion is a topological
stable spin texture and its spatiotemporal motion along the magnetic nano-track
intuitively interprets the pulse signal transmission between two interconnected
neurons. In such design, spike train in SNN could be encoded as particle-like
skyrmion train and further processed by the proposed skyrmion-synapse and
skyrmion-neuron within the same magnetic nano-track to generate output skyrmion
as post-spike. Then, both pre-neuron spikes and post-neuron spikes are encoded
as particle-like skyrmions without conversion between charge and spin signals,
which fundamentally differentiates our proposed design from other hybrid
Spin-CMOS designs. The system level simulation shows 87.1% inference accuracy
for handwritten digit recognition task, while the energy dissipation is ~1
fJ/per spike which is 3 orders smaller in comparison with CMOS based IBM
TrueNorth system
Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation
In the past years, Deep convolution neural network has achieved great success
in many artificial intelligence applications. However, its enormous model size
and massive computation cost have become the main obstacle for deployment of
such powerful algorithm in the low power and resource-limited mobile systems.
As the countermeasure to this problem, deep neural networks with ternarized
weights (i.e. -1, 0, +1) have been widely explored to greatly reduce the model
size and computational cost, with limited accuracy degradation. In this work,
we propose a novel ternarized neural network training method which
simultaneously optimizes both weights and quantizer during training,
differentiating from prior works. Instead of fixed and uniform weight
ternarization, we are the first to incorporate the thresholds of weight
ternarization into a closed-form representation using the truncated Gaussian
approximation, enabling simultaneous optimization of weights and quantizer
through back-propagation training. With both of the first and last layer
ternarized, the experiments on the ImageNet classification task show that our
ternarized ResNet-18/34/50 only has 3.9/2.52/2.16% accuracy degradation in
comparison to the full-precision counterparts
Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
Deep convolution neural network has achieved great success in many artificial
intelligence applications. However, its enormous model size and massive
computation cost have become the main obstacle for deployment of such powerful
algorithm in the low power and resource-limited embedded systems. As the
countermeasure to this problem, in this work, we propose statistical weight
scaling and residual expansion methods to reduce the bit-width of the whole
network weight parameters to ternary values (i.e. -1, 0, +1), with the
objectives to greatly reduce model size, computation cost and accuracy
degradation caused by the model compression. With about 16x model compression
rate, our ternarized ResNet-32/44/56 could outperform full-precision
counterparts by 0.12%, 0.24% and 0.18% on CIFAR- 10 dataset. We also test our
ternarization method with AlexNet and ResNet-18 on ImageNet dataset, which both
achieve the best top-1 accuracy compared to recent similar works, with the same
16x compression rate. If further incorporating our residual expansion method,
compared to the full-precision counterpart, our ternarized ResNet-18 even
improves the top-5 accuracy by 0.61% and merely degrades the top-1 accuracy
only by 0.42% for the ImageNet dataset, with 8x model compression rate. It
outperforms the recent ABC-Net by 1.03% in top-1 accuracy and 1.78% in top-5
accuracy, with around 1.25x higher compression rate and more than 6x
computation reduction due to the weight sparsity
Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search
Several important security issues of Deep Neural Network (DNN) have been
raised recently associated with different applications and components. The most
widely investigated security concern of DNN is from its malicious input, a.k.a
adversarial example. Nevertheless, the security challenge of DNN's parameters
is not well explored yet. In this work, we are the first to propose a novel DNN
weight attack methodology called Bit-Flip Attack (BFA) which can crush a neural
network through maliciously flipping extremely small amount of bits within its
weight storage memory system (i.e., DRAM). The bit-flip operations could be
conducted through well-known Row-Hammer attack, while our main contribution is
to develop an algorithm to identify the most vulnerable bits of DNN weight
parameters (stored in memory as binary bits), that could maximize the accuracy
degradation with a minimum number of bit-flips. Our proposed BFA utilizes a
Progressive Bit Search (PBS) method which combines gradient ranking and
progressive search to identify the most vulnerable bit to be flipped. With the
aid of PBS, we can successfully attack a ResNet-18 fully malfunction (i.e.,
top-1 accuracy degrade from 69.8% to 0.1%) only through 13 bit-flips out of 93
million bits, while randomly flipping 100 bits merely degrades the accuracy by
less than 1%
Blind Pre-Processing: A Robust Defense Method Against Adversarial Examples
Deep learning algorithms and networks are vulnerable to perturbed inputs
which is known as the adversarial attack. Many defense methodologies have been
investigated to defend against such adversarial attack. In this work, we
propose a novel methodology to defend the existing powerful attack model. We
for the first time introduce a new attacking scheme for the attacker and set a
practical constraint for white box attack. Under this proposed attacking
scheme, we present the best defense ever reported against some of the recent
strong attacks. It consists of a set of nonlinear function to process the input
data which will make it more robust over the adversarial attack. However, we
make this processing layer completely hidden from the attacker. Blind
pre-processing improves the white box attack accuracy of MNIST from 94.3\% to
98.7\%. Even with increasing defense when others defenses completely fail,
blind pre-processing remains one of the strongest ever reported. Another
strength of our defense is that it eliminates the need for adversarial training
as it can significantly increase the MNIST accuracy without adversarial
training as well. Additionally, blind pre-processing can also increase the
inference accuracy in the face of a powerful attack on CIFAR-10 and SVHN data
set as well without much sacrificing clean data accuracy
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack
Recent development in the field of Deep Learning have exposed the underlying
vulnerability of Deep Neural Network (DNN) against adversarial examples. In
image classification, an adversarial example is a carefully modified image that
is visually imperceptible to the original image but can cause DNN model to
misclassify it. Training the network with Gaussian noise is an effective
technique to perform model regularization, thus improving model robustness
against input variation. Inspired by this classical method, we explore to
utilize the regularization characteristic of noise injection to improve DNN's
robustness against adversarial attack. In this work, we propose
Parametric-Noise-Injection (PNI) which involves trainable Gaussian noise
injection at each layer on either activation or weights through solving the
min-max optimization problem, embedded with adversarial training. These
parameters are trained explicitly to achieve improved robustness. To the best
of our knowledge, this is the first work that uses trainable noise injection to
improve network robustness against adversarial attacks, rather than manually
configuring the injected noise level through cross-validation. The extensive
results show that our proposed PNI technique effectively improves the
robustness against a variety of powerful white-box and black-box attacks such
as PGD, C & W, FGSM, transferable attack and ZOO attack. Last but not the
least, PNI method improves both clean- and perturbed-data accuracy in
comparison to the state-of-the-art defense methods, which outperforms current
unbroken PGD defense by 1.1 % and 6.8 % on clean test data and perturbed test
data respectively using Resnet-20 architecture
KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
Deep Neural Networks (DNN) could forget the knowledge about earlier tasks
when learning new tasks, and this is known as \textit{catastrophic forgetting}.
While recent continual learning methods are capable of alleviating the
catastrophic problem on toy-sized datasets, some issues still remain to be
tackled when applying them in real-world problems. Recently, the fast
mask-based learning method (e.g. piggyback \cite{mallya2018piggyback}) is
proposed to address these issues by learning only a binary element-wise mask in
a fast manner, while keeping the backbone model fixed. However, the binary mask
has limited modeling capacity for new tasks. A more recent work
\cite{hung2019compacting} proposes a compress-grow-based method (CPG) to
achieve better accuracy for new tasks by partially training backbone model, but
with order-higher training cost, which makes it infeasible to be deployed into
popular state-of-the-art edge-/mobile-learning. The primary goal of this work
is to simultaneously achieve fast and high-accuracy multi task adaption in
continual learning setting. Thus motivated, we propose a new training method
called \textit{kernel-wise Soft Mask} (KSM), which learns a kernel-wise hybrid
binary and real-value soft mask for each task, while using the same backbone
model. Such a soft mask can be viewed as a superposition of a binary mask and a
properly scaled real-value tensor, which offers a richer representation
capability without low-level kernel support to meet the objective of low
hardware overhead. We validate KSM on multiple benchmark datasets against
recent state-of-the-art methods (e.g. Piggyback, Packnet, CPG, etc.), which
shows good improvement in both accuracy and training cost
Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
Deep Neural Network (DNN) trained by the gradient descent method is known to
be vulnerable to maliciously perturbed adversarial input, aka. adversarial
attack. As one of the countermeasures against adversarial attack, increasing
the model capacity for DNN robustness enhancement was discussed and reported as
an effective approach by many recent works. In this work, we show that
shrinking the model size through proper weight pruning can even be helpful to
improve the DNN robustness under adversarial attack. For obtaining a
simultaneously robust and compact DNN model, we propose a multi-objective
training method called Robust Sparse Regularization (RSR), through the fusion
of various regularization techniques, including channel-wise noise injection,
lasso weight penalty, and adversarial training. We conduct extensive
experiments across popular ResNet-20, ResNet-18 and VGG-16 DNN architectures to
demonstrate the effectiveness of RSR against popular white-box (i.e., PGD and
FGSM) and black-box attacks. Thanks to RSR, 85% weight connections of ResNet-18
can be pruned while still achieving 0.68% and 8.72% improvement in clean- and
perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its
PGD adversarial training baseline
A Progressive Sub-Network Searching Framework for Dynamic Inference
Many techniques have been developed, such as model compression, to make Deep
Neural Networks (DNNs) inference more efficiently. Nevertheless, DNNs still
lack excellent run-time dynamic inference capability to enable users trade-off
accuracy and computation complexity (i.e., latency on target hardware) after
model deployment, based on dynamic requirements and environments. Such research
direction recently draws great attention, where one realization is to train the
target DNN through a multiple-term objective function, which consists of
cross-entropy terms from multiple sub-nets. Our investigation in this work show
that the performance of dynamic inference highly relies on the quality of
sub-net sampling. With objective to construct a dynamic DNN and search multiple
high quality sub-nets with minimal searching cost, we propose a progressive
sub-net searching framework, which is embedded with several effective
techniques, including trainable noise ranking, channel group and fine-tuning
threshold setting, sub-nets re-selection. The proposed framework empowers the
target DNN with better dynamic inference capability, which outperforms prior
works on both CIFAR-10 and ImageNet dataset via comprehensive experiments on
different network structures. Taken ResNet18 as an example, our proposed method
achieves much better dynamic inference accuracy compared with prior popular
Universally-Slimmable-Network by 4.4%-maximally and 2.3%-averagely in ImageNet
dataset with the same model size
T-BFA: Targeted Bit-Flip Adversarial Weight Attack
Traditional Deep Neural Network (DNN) security is mostly related to the
well-known adversarial input example attack. Recently, another dimension of
adversarial attack, namely, attack on DNN weight parameters, has been shown to
be very powerful. As a representative one, the Bit-Flip-based adversarial
weight Attack (BFA) injects an extremely small amount of faults into weight
parameters to hijack the executing DNN function. Prior works of BFA focus on
un-targeted attack that can hack all inputs into a random output class by
flipping a very small number of weight bits stored in computer memory. This
paper proposes the first work of targeted BFA based (T-BFA) adversarial weight
attack on DNNs, which can intentionally mislead selected inputs to a target
output class. The objective is achieved by identifying the weight bits that are
highly associated with classification of a targeted output through a
class-dependent weight bit ranking algorithm. Our proposed T-BFA performance is
successfully demonstrated on multiple DNN architectures for image
classification tasks. For example, by merely flipping 27 out of 88 million
weight bits of ResNet-18, our T-BFA can misclassify all the images from 'Hen'
class into 'Goose' class (i.e., 100 % attack success rate) in ImageNet dataset,
while maintaining 59.35 % validation accuracy. Moreover, we successfully
demonstrate our T-BFA attack in a real computer prototype system running DNN
computation, with Ivy Bridge-based Intel i7 CPU and 8GB DDR3 memory
- …